<html>
<head><meta charset="utf-8"><title>high cpu usage · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html">high cpu usage</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="225879472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225879472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225879472">(Feb 10 2021 at 17:52)</a>:</h4>
<p>I have a multithreaded tcp server than receives binary data, parse it and creates some prometheus stats. One of the threads is clogging the cpu (stuck at 100% CPU usage). I did a perf recording and it looks like most of the CPU is consumed by prometheus lib (see bellow). The strange part is that there are no segments sent by the clients but the CPU usage is still 100%. Also I don't get anything if I strace the thread. So it looks stuck.</p>
<p>Any clue what could be happening? </p>
<div class="codehilite"><pre><span></span><code>    41.55%    41.10%           783  ebmd     ebmd               [.] prometheus::vec::MetricVec&lt;T&gt;::with_label_values
            |
            |--19.16%--0x1
            |          prometheus::vec::MetricVec&lt;T&gt;::with_label_values
            |
            |--16.93%--0
            |          |
            |          |--15.83%--prometheus::vec::MetricVec&lt;T&gt;::with_label_values
            |          |
            |           --0.70%--0
            |                     prometheus::vec::MetricVec&lt;T&gt;::with_label_values
            |
            |--4.71%--prometheus::vec::MetricVec&lt;T&gt;::with_label_values
            |
             --0.74%--ebmd::server::handle_ebm_stream_connection

    35.46%     0.00%             0  ebmd     [unknown]          [.] 0000000000000000
            |
            ---0
               |
               |--16.28%--prometheus::vec::MetricVec&lt;T&gt;::with_label_values
               |
               |--12.35%--hashbrown::map::make_hash
               |
               |--6.13%--&lt;std::collections::hash::map::DefaultHasher as core::hash::Hasher&gt;::write
               |
                --0.70%--0
                          prometheus::vec::MetricVec&lt;T&gt;::with_label_values

    20.24%    19.20%           344  ebmd     ebmd               [.] ebmd::server::handle_ebm_stream_connection
            |
             --19.80%--ebmd::server::handle_ebm_stream_connection

    19.16%     0.00%             0  ebmd     [unknown]          [.] 0x0000000000000001
            |
            ---0x1
               prometheus::vec::MetricVec&lt;T&gt;::with_label_values

    13.54%    12.75%           233  ebmd     ebmd               [.] hashbrown::map::make_hash
            |
            |--12.75%--0
            |          |
            |          |--11.56%--hashbrown::map::make_hash
            |          |
            |           --0.79%--&lt;std::collections::hash::map::DefaultHasher as core::hash::Hasher&gt;::write
            |
             --0.79%--hashbrown::map::make_hash

    10.79%    10.35%           194  ebmd     ebmd               [.] prometheus::atomic64::AtomicU64::inc_by_with_ordering
            |
             --10.64%--prometheus::atomic64::AtomicU64::inc_by_with_ordering

    10.67%    10.67%           216  ebmd     ebmd               [.] ebmd::parser::Parser::parse
            |
            ---ebmd::parser::Parser::parse

     6.47%     5.68%           115  ebmd     ebmd               [.] &lt;std::collections::hash::map::DefaultHasher as core::hash::Hasher&gt;::write
            |
            |--5.68%--0
            |          |
            |           --5.33%--&lt;std::collections::hash::map::DefaultHasher as core::hash::Hasher&gt;::write
            |
             --0.79%--&lt;std::collections::hash::map::DefaultHasher as core::hash::Hasher&gt;::write
</code></pre></div>



<a name="225880403"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225880403" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225880403">(Feb 10 2021 at 17:57)</a>:</h4>
<p>maybe your metric cardinality is huge and you're filling a MetricVec with new metrics as your code otherwise executes?</p>



<a name="225880635"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225880635" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225880635">(Feb 10 2021 at 17:58)</a>:</h4>
<p>Sounds like a question for <a href="https://github.com/tikv/rust-prometheus/issues">https://github.com/tikv/rust-prometheus/issues</a>. Looking at <a href="https://docs.rs/prometheus/0.11.0/src/prometheus/vec.rs.html#257-259">https://docs.rs/prometheus/0.11.0/src/prometheus/vec.rs.html#257-259</a>, I don't see anything obvious -- it might use CPU if <code>desc</code> is very large, but it should still finish.</p>



<a name="225880684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225880684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225880684">(Feb 10 2021 at 17:59)</a>:</h4>
<p>it is in the order of 20k. is that too much?</p>



<a name="225880772"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225880772" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225880772">(Feb 10 2021 at 17:59)</a>:</h4>
<p>I'd try to make a test case as a separate app</p>



<a name="225880900"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225880900" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225880900">(Feb 10 2021 at 18:00)</a>:</h4>
<p>20k doesn't sound much to me, but I'm not really familiar with metrics APIs</p>



<a name="225881017"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881017" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881017">(Feb 10 2021 at 18:01)</a>:</h4>
<p>It appears to be O(n) in the number of descriptors</p>



<a name="225881087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881087">(Feb 10 2021 at 18:01)</a>:</h4>
<p>Are you building it in release mode?</p>



<a name="225881387"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881387" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881387">(Feb 10 2021 at 18:03)</a>:</h4>
<p>I think so. Was actually thinking about that as it's build via Jenkins using docker. <br>
It looks like it's built in release mode by checking docker files. <br>
Is there a way to check the binary that it's built in release mode, to be sure?</p>



<a name="225881573"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881573" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881573">(Feb 10 2021 at 18:04)</a>:</h4>
<p>I don't know an easy way. You seem to have good symbol names, I'm guessing you also set <code>[profile.release] debug = true</code>?</p>



<a name="225881635"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881635" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881635">(Feb 10 2021 at 18:05)</a>:</h4>
<p>(That's not a bad thing to do, just checking.)</p>



<a name="225881782"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881782" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881782">(Feb 10 2021 at 18:06)</a>:</h4>
<p>I actually don't set that</p>



<a name="225881828"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881828" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881828">(Feb 10 2021 at 18:06)</a>:</h4>
<p>Hmm</p>



<a name="225881929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881929">(Feb 10 2021 at 18:07)</a>:</h4>
<p>when I build it locally on my PC the size in debug mode is much bigger than what I have in the docker container so it looks like it's using release mode</p>



<a name="225881965"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225881965" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225881965">(Feb 10 2021 at 18:07)</a>:</h4>
<p>That also depends on whether you're stripping the binary or not, and on the platform</p>



<a name="225882078"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225882078" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225882078">(Feb 10 2021 at 18:08)</a>:</h4>
<p>E.g. <code>rust-analyzer</code> Windows binaries are 50% larger than Linux ones, all other things being equal</p>



<a name="225882126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225882126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225882126">(Feb 10 2021 at 18:09)</a>:</h4>
<p>You could enable <code>debug</code>, that way <code>perf</code> might give you more info if the issue happens again</p>



<a name="225882170"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225882170" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225882170">(Feb 10 2021 at 18:09)</a>:</h4>
<p>But I'd still try to reproduce it in a test program</p>



<a name="225884826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225884826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225884826">(Feb 10 2021 at 18:29)</a>:</h4>
<p>thanks for the suggestions</p>



<a name="225889969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225889969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225889969">(Feb 10 2021 at 19:00)</a>:</h4>
<p>checking the instructions, the CPU is used in instructions prefixed with LOCK</p>
<div class="codehilite"><pre><span></span><code>Percent│125:   mov       %rdi,0x10(%rsp)
       │       mov       (%r12),%rax
       │       test      $0x8,%al
       │     ↓ jne       258
  1.34 │       mov       %rax,%rcx
       │       add       $0x10,%rcx
       │     ↓ jb        258
 22.85 │       lock      cmpxchg   %rcx,(%r12)
       │     ↓ jne       258
  2.16 │14f:   mov       %r12,(%rsp)
       │       lea       0x18(%r13),%rdi
       │       lea       0x10(%rsp),%rsi
       │       mov       %rdi,0x8(%rsp)
  1.34 │     → callq     hashbrown::map::make_hash
  0.78 │       mov       0x28(%r13),%rdx
  0.22 │       mov       0x30(%r13),%r11
  0.11 │       mov       %rdx,%rsi
  0.45 │       and       %rax,%rsi
  0.45 │       shr       $0x39,%rax
       │       lea       0x10(%rsi),%rdi
  0.34 │       and       %rdx,%rdi
  0.78 │       movdqu    (%r11,%rsi,1),%xmm1
  1.01 │       movd      %eax,%xmm0
       │       punpcklbw %xmm0,%xmm0
  0.11 │       pshuflw   $0xe0,%xmm0,%xmm0
  0.89 │       pshufd    $0x0,%xmm0,%xmm0
  0.56 │       movdqa    %xmm0,%xmm2
  0.68 │       pcmpeqb   %xmm1,%xmm2
       │       pmovmskb  %xmm2,%eax
  0.11 │       mov       $0x10,%r9d
  0.34 │       test      %ax,%ax
       │     ↓ je        216
  0.57 │1ae:   bsf       %ax,%bp
  0.11 │       movzwl    %bp,%ebx
  0.56 │       add       %rsi,%rbx
  0.22 │       mov       0x10(%rsp),%r10
  0.45 │       and       %rdx,%rbx
       │       mov       %rbx,%rbp
  0.45 │       shl       $0x4,%rbp
       │       mov       %r11,%rcx
  0.38 │       sub       %rbp,%rcx
  2.04 │       cmp       -0x10(%rcx),%r10
       │     ↓ jne       281
       │       neg       %rbx
       │1da:   shl       $0x4,%rbx
       │       mov       -0x8(%r11,%rbx,1),%rbp
 17.91 │       lock      addq      $0x1,0x0(%rbp)
       │     ↓ jle       5ee
       │       test      %rbp,%rbp
       │     ↓ je        2f9
  0.78 │       mov       $0xfffffffffffffff0,%rax
 23.41 │       lock      xadd      %rax,(%r12)
  0.11 │       and       $0xfffffffffffffff2,%rax
</code></pre></div>



<a name="225893518"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225893518" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225893518">(Feb 10 2021 at 19:24)</a>:</h4>
<p>That might be the <code>prometheus::atomic64::AtomicU64::inc_by_with_ordering</code> in the tree you posted earlier.</p>



<a name="225893946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225893946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225893946">(Feb 10 2021 at 19:27)</a>:</h4>
<p>You can get <code>perf</code> to show the corresponding source code too if you compile locally with debug info.</p>



<a name="225894417"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/high%20cpu%20usage/near/225894417" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/high.20cpu.20usage.html#225894417">(Feb 10 2021 at 19:31)</a>:</h4>
<p><span class="user-mention silent" data-user-id="363601">Maximilian Hristache</span> <a href="#narrow/stream/122651-general/topic/high.20cpu.20usage/near/225879472">said</a>:</p>
<blockquote>
<p>Also I don't get anything if I strace the thread. So it looks stuck.</p>
</blockquote>
<p>It could be a logic bug where it loops forever. The innermost loop can be a red herring if it's an outer one that doesn't make progress.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>